New advances in fluorescent probe technologies and imaging techniques are enabling the simultaneous imaging of many more biological entities (classes) than has been possible before. For example, spectral imaging (CRI Inc. http://www.cri-inc.com/instruments/products/index.shtml), or emission finger-printing (Carl Zeiss LSM 510 META. http://www.zeiss.de/C12567BE0045ACF1/Inhalt-Frame/C98CD5EF1EFAF4EEC1256AC5003148E9) acquire images at multiple wavelengths and can generate precise optical spectra at every pixel. In a second step, this information is used for the digital separation of up to eight fluorophores. Spectral karyotyping (SKY) (Schrock E, du Manoir S, Veldman T, Schoell B, Wienberg J, Ferguson-Smith M A, Ning Y, Ledbetter D H, Bar-Am I, Soenksen D, Garini Y, Ried T. 1996. Multicolor spectral karyotyping of human chromosomes. Science. 26; 273(5274):494-7) and multiplexed fluorescence in situ hybridization (MFISH) (Speicher M R, Gwyn Ballard S, Ward D C. 1996. Karyotyping human chromosomes by combinatorial multi-fluor FISH. Nat Genet. 12(4):368-75.) enable the simultaneous visualization of the endogenous arrangement of the complete karyotype. These innovations increase the amount of biology we can resolve in an image by a factor of ten. Though the technologies underlying these innovations differ, the implications for biological image analysis are the same; current and next generation biological images will contain many more biological objects and relations, and many more classes of objects than has been the case until now. There will be corresponding growth in the need among scientists in basic research, drug discovery and diagnostic imaging for a versatile and flexible tool to assist users for the detection and analysis of patterns in the relational, spectral, temporal arrangement of these individual objects and object classes.
Previously limited to three (usually one channel for the nuclear background, and a channel each for two chromosome classes), cytogeneticists can now see an entire karyotype simultaneously in a single SKY or MFISH image. FIGS. 1 and 2 demonstrate this profound advance. FIG. 1 shows a typical standard FISH image of the interphase nucleus. There are three fluorescent images, FIG. 1A, FIG. 1B and FIG. 1C. Object mask images labeled by class can be produced from each channel image through image processing. See FIG. 1D, FIG. 1E and FIG. 1F. Here there are three object classes; the nuclear envelope class containing one individual object (FIG. 1A and FIG. 1D), a chromosome class containing two individual chromosomes (homologs), FIG. 1B and FIG. 1E, and a second, different chromosome class containing two individual homologs (FIG. 1C and FIG. 1F).
In contrast, FIG. 2 shows a SKY image of the metaphase rosette. SKY makes use of an interferogram to process multiple spectra at the pixel level, and algorithmically combines that information with a CCD image (FIG. 2A) to produce an output image composed of N individually segmented object masks (FIG. 2B), where N corresponds to roughly two times the number of chromosomes in the karyotype (Schrock E, du Manoir S, Veldman T, Schoell B, Wienberg J, Ferguson-Smith M A, Ning Y, Ledbetter D H, Bar-Am I, Soenksen D, Garini Y, Ried T. 1996. Multicolor spectral karyotyping of human chromosomes. Science. 26; 273(5274):494-7; Applied Spectral Imaging. http://www.spectral-imaging.com). In FIG. 2A, N=40 for these mouse cells (humans would have 46). Each mask is labeled via a look up table with a user defined RGB value corresponding to the object (chromosome) class. In FIG. 2B there are two individual objects per object class.
Despite these advances, little software has been developed for the analysis of relational patterns among biological objects. The majority of today's analysis software is designed to measure the response of a biological system to perturbation (Giuliano K, Kapur R. 2002. System for cell based screening. U.S. Pat. No. 6,416,959; Harris T D, Hansen R L, Karsh W, Nicklaus N A, Trautman J K. 2002. Method and apparatus for screening chemical compounds. U.S. Pat. No. 6,388,788.), or to manually or automatically score disease samples (TriPath Imaging Inc., FocalPoint Slide Profiler. http://www.tripathimaging.com/usproducts/focalpoint.htm; Applied Imaging Corp., CytoVision. http://www.appliedimagingcorp.com/usa/cyto/index.htm.) by analyzing the fluorescently labeled biology in the image. In stark contrast to this type of evaluation of phenotypic characteristics of biological objects as an indicator in modern biology, software can assist human to analyze relational patterns in the location or relational, temporal arrangement of biological objects is virtually inexistence in the life sciences applications.
Due to the tedious nature of manual analysis as well as the lack of automatic analysis technology, scientists are using application-specific image analysis methods that generate only a very few patterns and few samples for analysis. The inability to rapidly create and analyze a large number of relational patterns makes it inefficient to find important characteristics. The domain of possible relational patterns and combinations of patterns is large, and even assuming that one pattern will reveal a preference of relational arrangement (which indeed may not be the case), the possibility that any one pattern will reveal that preference is low. Furthermore, current methods suffer from low repeatability because they are based on imprecise image processing techniques developed for specific applications. These methods fail in the face of typical variations found in biological images such as large variations in the shape, orientation and size of biological objects (even among the same class), image variations resulting from operator or equipment variability, and variation in image orientation.
It would be ideal for scientists to possess an efficient, robust, accurate and flexible tool for pattern creation and review. Such a tool would enable them to distinguish the difference between a pattern signal and noise, and quickly find interesting relational patterns in biological images. The pattern sets could be scientist-generated rather than computer generated, this allows the analysis outcome to be easily validated.
Scientists have long been interested in the relational arrangement of biological objects. The idea that chromosomes may be arranged in a specific fashion has been considered since the time of Boveri (Baltzer F. 1964. Theodor Boveri. Science. 15(144):809-15). A well-known demonstration of general chromosomal organization is the Rabl orientation, a polarization of centromeres and telemeres, observed in the early embryo of the fruit fly. Patterns in the relational arrangement of biological objects have been studied outside the nucleus as well in skin cancer diagnosis (BC Cancer Research Centre. Research Arm of the BC Cancer Agency. http://www.bccrc.ca/ci/ta01_archlevel.html), retinal cell arrangement (Eglen S J, Raven M A, Tamrazian E, Reese B E. 2003. Dopaminergic amacrine cells in the inner nuclear layer and ganglion cell layer comprise a single functional retinal mosaic. J Comp Neurol. 446(3):343-55.), fungal spores arrangement (Jones C L, Lonergan G T, Mainwaring D E. Minimal spanning tree analysis of fungal spore relational patterns. Published online at http://www.swin.edu.au/chem/bio/fractals/mst01.htm) and platelets in wound healing (Beals M, Gross L, Harrell S. 2000. Cell aggregation and sphere packing. The Institute for Environmental Modeling at the University of Tennessee).
Prior art detection and analysis of relational arrangement patterns is a two step process that requires image processing software, such as Universal Imaging's Metamorph (Universal Imaging Corp. Metamorph. http://www.image1.com/products/metamorph/) or Media Cybernetics' ImagePro (Media Cybernetics Inc. ImagePro. http://www.mediacy.com/famip.htm), to perform image segmentation and generate pattern measurements, and a data mining or statistics package such Spotfire's DecisionSite (Ahlberg C, Truve S, Wistrand E. 2000. System and method for automatic analysis of data bases and for user controlled dynamic querying. U.S. Pat. No. 6,014,661; Spotfire Inc. DecisionSite. http://www.spotfire.com/products/decision.asp) or Insightful's S-PLUS (Insightful S-PLUS. http://www.insightful.com/products/splus/default.asp) to evaluate the complex pattern signal. Even a single pattern signal can be complex because of the large number of object interactions. For example analyzing a nearest neighbor pattern across the objects in FIG. 2 would require the analysis of 1,560 interactions (the nearest neighbor pattern for relation i×j is not equivalent for j×i). Since there is no integrated approach, it is difficult for scientists to iterate quickly between pattern creation and pattern analysis. This approach is far from general purpose, and inefficient for finding patterns.
Some examples of this type of prior art approach:
Eglen et al. (Eglen S J, Raven M A, Tamrazian E, Reese B E. 2003. Dopaminergic amacrine cells in the inner nuclear layer and ganglion cell layer comprise a single functional retinal mosaic. J Comp Neurol. 446(3):343-55.) used Voronoi domains to analyze the relational arrangement of dopaminergic amacrine cells in two different layers of the ferret retina to determine if the arrangement was different between layers, which might indicate distinct cell function for the cells in the different layers. Eglen calculated a regularity index statistic for each cell as the Voronoi domain area divided by the standard deviation of all Voronoi domain areas in the image.
The BC Cancer Research Centre uses Minimum Spanning Tree (MST) vertex length as a feature on which statistical analysis is performed (BC Cancer Research Centre. Research Arm of the BC Cancer Agency. http://www.bccrc.ca/ci/ta01_archlevel.html). These outputs are then linearly combined into an index that can be used for the quantitative diagnosis of pre-neoplastic lesions. Jones et al. (Jones C L, Lonergan G T, Mainwaring DE. Minimal spanning tree analysis of fungal spore relational patterns. Published online at http://www.swin.edu.au/chem/bio/fractals/mst01.html) also used MST to look at fungal spore relational arrangement and its relation to the asexual reproduction mechanism. Research at the Institute of Environmental Modeling, University of Tennessee has utilized sphere packing theory to quantify the density of aggregated cells involved in wound healing (Beals M, Gross L, Harrell S. 2000. Cell aggregation and sphere packing. The Institute for Environmental Modeling at the University of Tennessee.)
Basic research in chromosome arrangement in the field of cytogenetics has been rejuvenated and accelerated with recent discoveries that link the developmental regulation of lineage-restricted genes with their nuclear compartmentalization (Brown KE, Guest SS, Smale ST, Hahm K, Merkenschlager M, Fisher AG. 1997. Association of transcriptionally silent genes with Ikaros complexes at centromeric heterochromatin. Cell. 91(6):845-54; Brown K E, Baxter J, Graf D, Merkenschlager M, Fisher AG. 1999. Dynamic repositioning of genes in the nucleus of lymphocytes preparing for cell division. Mol Cell. 3(2):207-17; Kosak S T, Skok J A, Medina K L, Riblet R, Le Beu M M, Fisher AG, Sing H. 2002. Subnuclear compartmentalization of immunoglobulin loci during lymphocyte development. Science. 296(5565): 158-62; Schubeler D, Francastel C, Cimbora D M, Reik A, Martin D I, Groudine M. 2000. Nuclear localization and histone acetylation: a pathway for chromatin opening and transcriptional activation of the human beta-globin locus. Genes Dev. 14(8):940-950.) The implication of these findings is that during cellular differentiation, the nucleus is reorganized in a way that permits the regulation of all relevant genes for a particular cell type. Similarly, there is an implication that disease progression also has an impact on nuclear organization and gene regulation. As a result, leading cytogenetics researchers are attempting to find and understand patterns in chromosome relational arrangements in an ad hoc fashion, developing their own features for use with standard FISH images (Croft J A, Bridger J M, Boyle S, Perry P, Teague P, Bickmore W A. 1999. Differences in the localization and morphology of chromosomes in the human nucleus. J. Cell Biol. 145(6):1119-31.; Boyle S, Gilchrist S, Bridger J M, Mahy N L, Ellis J A, Bickmore W A. 2001. The relational organization of human chromosomes within the nuclei of normal and emerin-mutant cells; Bridger J M, Boyle S, Kill IR, Bickmore W A. 2000. Re-modeling of nuclear architecture in quiescent and senescent human fibroblasts. Curr Biol. 10(3): 149-52.; Nagele R, Freeman T, McMorrow L, Lee H V. 1995. Precise relational positioning of chromosomes during prometaphase: evidence for chromosomal order. Science. 270(5243):1831-5.; Allison D C, Nestor A L. 1999. Evidence for a relatively random array of human chromosomes on the mitotic ring. J. Cell Biol. 145(1):1-14.). For example, Misteli has used nearest neighbor (Parada L A, McQueen P G, Munson P J, Misteli T. 2002. Conservation of relative chromosome positioning in normal and cancer cells. Curr Biol. 12(19):1692-7.), Allison has used radial angle between chromosomes (Allison D C, Nestor A L. 1999. Evidence for a relatively random array of human chromosomes on the mitotic ring. J. Cell Biol. 145(1): 1-14) and Bickmore (Croft J A, Bridger J M, Boyle S, Perry P, Teague P, Bickmore W A. 1999. Differences in the localization and morphology of chromosomes in the human nucleus. J. Cell Biol. 145(6):1119-31) has used distance from the boundary of the nucleus.
These examples demonstrate that scientists are using application specific image analysis approaches that generate only a few patterns (e.g. Voronoi based regularity index, MST vertex length, nearest neighbor etc.) for analysis. But unfortunately, reliance on a single pattern or small pattern set reduces the chance that important patterns will be detected.
Biological samples such as SKY images of the metaphase rosette (FIG. 2A) are non-standard, and often are distorted and arbitrarily oriented. It is difficult to process a large number of SKY images in a robust and accurate fashion that allows result accumulation, which is critical for population analysis to confirm hypotheses or discover subtle pattern differences. The prior art image processing software is unable to handle these large variations. It is impractical to manually normalize the samples. It is highly desirable to have an image analysis tool to automatically normalize the distortion and inter-sample variations among input images for robust and accurate, automated measurements across multiple samples.
A key limitation of current approaches to relational pattern detection is the lack of comprehensive relational pattern configuration features for use in pattern detection. This is due to the tedious nature of manual analysis as well as a lack of image analysis technology. This limitation prevents the detection of subtle differences or higher order (non-trivial) relations within a class or between classes. By using more relational pattern features, users will be more likely to tell the difference between a pattern's signal and noise and could find meaningful patterns efficiently. Also, leading laboratories have indicated that pattern features must be easily understandable so that outcomes can be validated.
Scientists need an integrated image processing and data analysis tool that can enable the creation of many user defined relational pattern features, and support interactive feature mining for pattern detection and analysis; all in a package that is easy to use and requires no programming. No such tool or combination of tools currently exists.